Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Honeybadger.lambdaHandler() to return async handler #680

Merged
merged 24 commits into from
Feb 2, 2022

Conversation

subzero10
Copy link
Member

@subzero10 subzero10 commented Dec 30, 2021

Status

READY

Description

This PR changes the result of Honeybadger.lambdaHandler() to be a promise based (async) handler instead of the outdated callback based handler.

Discussion: #674
Issue: #677

Todos

  • Return async handler instead of call back based
  • Add proper type definitions to the handler (from @types/aws-lambda)
  • Add example project for AWS Lambda
  • Honeybadger.notifyAsync()
  • Tests - Make sure new and existing tests pass
  • Changelog Entry (unreleased)
  • Test manually with example project
  • AsyncLocalStorage (or something equivalent) Separate issue: AWS Lambda - Isolated context per handler #688
  • After release: Modify examples/aws-lambda/package.json to install @honeybadger-io/js latest release (it currently installs from aws_lambda_async_#677)
  • After merge: Update AWS projects to point to aws lambda example. Discussion here.

Steps to Test or Reproduce

See examples/aws-lambda/README.md

@subzero10 subzero10 self-assigned this Dec 30, 2021
@subzero10
Copy link
Member Author

I know we haven't reached to a conclusion on this, but based on the implementations of Sentry and Bugsnag, I decided to drop the usage of domain and simply opt for a try/catch block. What do you think? @shalvah @joshuap

@shalvah
Copy link
Contributor

shalvah commented Dec 31, 2021

Well done @subzero10 . I've got some thoughts, but I'm currently on holiday, so it'll have to be next week

package.json Show resolved Hide resolved
src/server/aws_lambda.ts Outdated Show resolved Hide resolved
@shalvah
Copy link
Contributor

shalvah commented Dec 31, 2021

I agree with you that a try/catch is enough, but I wonder about context (which I think is what domains were used for). For instance, in this handler.js:

const hb = require('honeybadger');

module.exports.handler1 = () => {
  hb.context({a: "1"});
});

module.exports.handler1 = () => {
  hb.context({a: "2"});
});

can we be sure these context calls won't clash if both these functions get executed, considering the same Lambda instance might be reused? I think we should add a test for that. If so, we could (should) use AsyncLocalStorage and polyfills to fix it.

@subzero10
Copy link
Member Author

I agree with you that a try/catch is enough, but I wonder about context (which I think is what domains were used for). For instance, in this handler.js:

const hb = require('honeybadger');

module.exports.handler1 = () => {
  hb.context({a: "1"});
});

module.exports.handler1 = () => {
  hb.context({a: "2"});
});

can we be sure these context calls won't clash if both these functions get executed, considering the same Lambda instance might be reused? I think we should add a test for that. If so, we could (should) use AsyncLocalStorage and polyfills to fix it.

I was worried that there would be a performance penalty with AsyncLocalStorage, but it seems this is not the case anymore:

Still, I am wondering if we could just use a dictionary and store honeybadger context using key context.awsRequestId. Wouldn't that be enough?

@shalvah
Copy link
Contributor

shalvah commented Jan 6, 2022

The dictionary approach should work, but wouldn't that possibly lead to a memory leak? We'll keep adding to the dictionary and never remove unless there's a crash. I don't know how long Lambda instances stick around for, though...🤔

@subzero10
Copy link
Member Author

The dictionary approach should work, but wouldn't that possibly lead to a memory leak? We'll keep adding to the dictionary and never remove unless there's a crash. I don't know how long Lambda instances stick around for, though...🤔

Good question. I wonder if there's a way to find that out. I will do some googling.

Also, question that came up now: Even if the lambda instance is re-used, would it be the same nodejs process? If not, we wouldn't need to do anything here right?

@shalvah
Copy link
Contributor

shalvah commented Jan 6, 2022

Yeah, if it's a different process,, we would be safe. But considering that the reused instance already has the state initialised, I'm leaning towards same process.

@shalvah
Copy link
Contributor

shalvah commented Jan 6, 2022

Hmm. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html#runtimes-lifecycle-shutdown

Screenshot_20220106-115845~2

Copy link
Member

@joshuap joshuap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the code, looking good. I added a few questions. @subzero10 are you still having issues with the setTimeout errors? I think that may be related to why we were using domains.

examples/aws-lambda/README.md Show resolved Hide resolved
src/server/aws_lambda.ts Outdated Show resolved Hide resolved
@subzero10
Copy link
Member Author

subzero10 commented Jan 9, 2022

I read through the code, looking good. I added a few questions. @subzero10 are you still having issues with the setTimeout errors? I think that may be related to why we were using domains.

Yes, I am getting this error when throwing an uncaught error in a setTimeout:

Unknown application error occurred

I'm still investigating.
Update: It seems that AWS Lambda has a default listener that exits the process before it reaches our uncaughtException listener. By removing the first uncaughtException listener (only when running in AWS Lambda), errors are caught in our listener and reported to Honeybadger.

For the rest, there were times where I used to get this error:

Error report failed: an unknown error occurred. message=Client network socket disconnected before secure TLS connection was established

I think it was because I was throwing the error (therefore ending the aws lambda process) before the http request to Honeybadger is completed.
I updated the code to return a promise (that rejects) inside the catch block: https://github.com/honeybadger-io/honeybadger-js/pull/680/files#diff-5d4654b27960ff4b6441a43aa555e9be5ba6dfca79303d6abd464a077e38bcf2R49

@subzero10
Copy link
Member Author

can we be sure these context calls won't clash if both these functions get executed, considering the same Lambda instance might be reused? I think we should add a test for that. If so, we could (should) use AsyncLocalStorage and polyfills to fix it.

@shalvah @joshuap

Should we implement this in this PR? I'm thinking that this potential issue already exists with the current lambdaHandler implementation, so I'm wondering if AsyncLocalStorage is out of the scope of this PR: callback based to promise based lambda handler.
I'm only suggesting a separate PR/ticket because I like PRs to be as small as possible.

@subzero10
Copy link
Member Author

subzero10 commented Jan 11, 2022

can we be sure these context calls won't clash if both these functions get executed, considering the same Lambda instance might be reused? I think we should add a test for that. If so, we could (should) use AsyncLocalStorage and polyfills to fix it.

@shalvah @joshuap

Should we implement this in this PR? I'm thinking that this potential issue already exists with the current lambdaHandler implementation, so I'm wondering if AsyncLocalStorage is out of the scope of this PR: callback based to promise based lambda handler. I'm only suggesting a separate PR/ticket because I like PRs to be as small as possible.

I tried to start work on AsyncLocalStorage, but it seems that it won't be that simple to implement (or maybe I should say there will be enough changes to have on a separate PR). We will have to do some refactoring to externalize the store for context and breadcrumbs.

@subzero10 subzero10 requested review from shalvah and joshuap January 11, 2022 20:43
@shalvah
Copy link
Contributor

shalvah commented Jan 14, 2022

Will review over the weekend @subzero10 . Sorry for the delay.

@subzero10 subzero10 mentioned this pull request Jan 15, 2022
3 tasks
Copy link
Contributor

@shalvah shalvah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @subzero10, just a few concerns.

src/server/aws_lambda.ts Outdated Show resolved Hide resolved
src/server/aws_lambda.ts Outdated Show resolved Hide resolved
Copy link
Member

@joshuap joshuap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed everything so far and am following along. The code looks good to me, though I think @shalvah is more in the headspace of the problem. I'll take another look once #684 gets merged in.

@subzero10
Copy link
Member Author

subzero10 commented Jan 23, 2022

@shalvah @joshuap I think this is ready. I created this issue to tackle context isolation. I already had a solution in mind about this, but now that we return both async and sync versions of the handler I will have to think about it again 😅

@subzero10 subzero10 requested review from joshuap and shalvah January 23, 2022 09:48
@subzero10
Copy link
Member Author

@joshuap @shalvah I added notifyAsync documentation, see PR here.

Copy link
Contributor

@shalvah shalvah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I'll try to take it for a spin on a project soon.

@subzero10
Copy link
Member Author

@joshuap I think this is safe to merge and release, unless you have any other feedback.

@joshuap joshuap merged commit 7ea5fb7 into master Feb 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants